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Purpose 


Army  Need:  To  identify  initial  entry  personnel  who  have  the  potential 
for  high  performance  in  the  Cyberspace  Workforce 

Objective:  Develop  and  refine  selection  measures  for  new  accessions 
to  more  fully  assess  an  individual’s  potential  and  more  accurately 
predict  performance,  behavior,  and  attitudes. 

Efforts:  Four  research  efforts  benefitting  the  selection  and 
classification  of  new  accessions  for  cyber  occupations  will  be 
discussed: 

•  Cyber  Test  Validation  Research  for  the  Joint  Cyber  Analysis 
Course 

•  Cyber  Test  Validation  Research  within  a  sample  of  Information 
Technology  Specialists 

•  Common  Cyber  Capabilities  Test 

•  Systems  Thinking  Ability  Test  2 


Cyber  Test  (CT) 


Army  Need:  To  identify  initial  entry  and  in-service  personnel  who  have  the  potential  for 
high  performance  in  the  Cyberspace  Workforce 

In  2006  an  ASVAB  review  panel  provided  recommendations  for  enhancing  the 
ASVAB:  “...research  should  be  conducted  to  develop  and  evaluate  a  test  of  information 
and  communications  technology  literacy.  ” 

In  2008  the  Air  Force  developed  the  CT  test  with  consultation  from  SMEs  from  all  U.S. 
services 


uyoer  i est  (uj  -  formerly  Known  as  iu i  l 


IT  Software/Tools  and  PC  Configuration/Maint.  (36%) 

What  kind  of  software  (spreadsheet/word  processing)  is 
appropriate  for  a  specified  task? 

Networking  and  Communications  (27%) 

•  What  are  IP  addresses  ?  What  are  network  protocols  ? 

Security  and  Compliance  (23%) 

What  are  viruses?  What  are  security  concerns  with  the 

Internet? 

Software  Programming  and  Web  Development  (14%) 

•  What  are  differences  between  different  data  formats  ? 

Overview  of  the  Cyber  Test: 

•  29  multiple  choice  items. 

•  1 5-20  minutes  to  complete  the  test. 

•  Two  different  test  forms  currently  in  use. 

•  The  CT  is  a  special  test  that  applicants  take. 

•  Future  joint  service  research  may  develop  a  computerized  adaptive  test  version 
of  the  CT. 


Joint  Cyber  Analysis  Course  (JCAC) 
Cyber  Test  Validation  Research 


Operational  use  of  CT  &  ASVAB-Skilled  Technical  (ST)  for  two  cyber  jobs: 

•  Administered  CT  operationally  since  2  June  2014 

•  Required  minimum  ST  of  112  and  CT  of  60 


Validation  Procedure:  (Administered  via  Paper  &  Pencil) 


Soldiers  going  into 
these  2  cyber  jobs 
attend  JCAC 


Performance  outcomes 

Link  test  scores  to 

administered  during  last 

training  performance, 

module  of  JCAC  or  after 

attitudes,  and  attrition 

JCAC  completion 

outcomes 

Restarts  -  Recycles 

Measures  for  Validation 


Tests: 

•  ICTL  /  Cyber  Test 

•  ASVAB  -  Skilled  Technical 


Outcomes: 

•  JCAC  grades 

•  Performance  rating  scales 

•  Job  knowledge  test 

•  Attitudes 

•  Attrition 


Data  Collection  Update: 


Males:  87 
Females:  10 
Total:  97 


Note:  ARI  has  recently  developed  two  in-service  versions  of  the  CT  that  are  parallel  to 
the  CT  used  for  applicants. 


Validation  Procedure 


Past  Cyber  Test  Validation 

Research 


Military 
ntelligence 


CT  test  administered 
during  first  week  of 
advanced  training 


Restarts  -  Recycles 


Performance  outcomes 
collected  from 
administrative  databases 


Performance  outcomes 
administered  during  last 
week  of  advanced  training 


Link  CT  and  other  test 
scores  to  performance, 
attitudes  (Signal  only), 
and  attrition  outcomes 


7" 


*Outcomes  of  Interest 

Attrition:  Administrative  data  extracted  from  personnel  database 
Training  Failure/Restarts:  Administrative  data  extracted  from  Training  System 
AIT  Course  Grade  Average:  Administrative  data  extracted  from  Training  System 
Jlob  Fit  (Signal  only):  Measures  the  Soldier's  perceived  fit  with  their  job 

Peer  Rated  Training  Performance  (Signal  only):  Evaluates  Soldier  performance  across  a  number  of 
identified  job-specific  performance  domains.  Developed  with  SMEs  in  respective  jobs. 


Training  Performance  Domains 


Learns  to  implement  a  network 

Learns  hardware  concepts 

Learns  to  administer  software  applications 

Learns  the  fundamentals  of  network  security 

Learns  basics  of  troubleshooting 

Learns  safety  procedures 


*  Significant  positive  relationships 
were  found  between  the  CT  scores 
and  these  outcome  variables  within  a 
sample  of  Information  Technology 
Specialists. 
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CT/ICTL  Validity  Research  Results 


CA 


Performance  Rating  Scale  Mean 

N  =  2,139,  r  =  .327 


4.32 


Advanced  Individual  Training  Final  Course 

Grade 

N=  1,805,  r=. 405 
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Job  Knowledge  Test 
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Knowledge-based  Measures 


Pros 

•  High  fidelity  to  cyber  realm 

•  Identifies  a  pre-existing  interest  in  cyber 
/  drawn  to  engage  with  cyber 

•  Straightforward  to  create  &  interpret 

•  Adds  additional  predictability  above  and 
beyond  the  ASVAB,  especially  in  high 
ability  populations 

•  Preliminary  evidence  suggests  that  the 
CT  has  lower  adverse  impact  than  most 
ASVAB  sub-tests 


Cons 

•  Technology  changes  quickly;  content 
becomes  obsolete 

•  As  a  result,  knowledge-based 
measures  require  more  cost  and  effort 
to  maintain  over  time 

•  Might  miss  high-potential  applicants 
without  prior  technology  exposure 

•  Exposure  of  items  over  time  may 
increase  the  risk  of  compromising  item 
integrity 


Two  projects  are  underway  to  examine  alternatives  to 
knowledge-based  tests: 

1 .  Common  Cyber  Capabilities  Test 

2.  Systems  Thinking  Assessment 
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Common  Cyber  Capabilities  Test  (C^3) 


Common  Cyber  Capabilities  Test  -  Assessment  of  the  skills,  abilities,  and  other 
characteristics  that  are  most  predictive  of  trainability  and  success  across  the 
Army’s  various  cyber  jobs.  CA3  will  be  a  computer-administered  self-scoring  test  of 
skills  and  abilities  that  are  not  covered  by  established  Army  measures. 


3.  Develop  measures  of  selected 
capabilities 

O 

w  4.  Conduct  validation  of  items  and  scales 

z 

< 

oJ  5.  Combine  developed  and  validated 
measures  into  one  cohesive  product 

6.  Validate  to  Army  cyber  populations 


For  each  cyber  job  we  will: 

1 .  Solicit  assistance  from  potential 
sponsors 

2.  Review  available  task  and 
knowledge/skills/abilities  lists 

3.  Meet  with  Subject  Matter  Experts  to 
identify  capabilities  needed  in  their  jobs 

4.  Meet  with  course  instructors  to  identify 
which  capabilities  best  predict  trainability 

5.  Solicit  leadership’s  feedback  on  lists 
yielded  by  steps  2-4 


Lists  for  each  job  will  be  combined  to 
select  the  5  to  7  capabilities  that  will  be 
measured  by  the  CA3 


Systems  Thinking  Assessment 


Systems:  Dynamic  sets  of  interconnected  elements  organized  in  a  coherent 
way  that  achieve  a  function  or  purpose 


Systems  Thinking:  Understanding  activity  and  changing  characteristics  of  a 
system  as  it  operates  and  the  ability  to  anticipate  effects  of  actions  upon  the 
system  to  attain  particular  outcome 


Systems 
Thinking 
Ability 
Antecedents 


Figure.  Preliminary  Model  of  Systems  Thinking  Ability 
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Planned  Completed 


Systems  Thinking  Ability  Test  (STA) 


ARI  has  been  researching  a  distinct  approach  to: 

•  Identify  component  capabilities  and  model  of  systems  thinking  ability 

•  Develop  and  validate  measures  of  component  capabilities 
•  Will  look  to  solicit  SME  feedback  on  initial  measures 

•  Lay-out  how  measures  could  be  combined  into  one  cohesive  computer- 
based  instrument  of  systems  thinking 

Research  Plan 

1 .  Develop  and  validate  assessments  for  five 
STA-related  abilities 

2.  Develop  the  conceptual  design  and  proof-of- 
concept  for  a  systems  thinking  assessment 
game 

3.  Develop  an  operational  version  of  the  STA 
game  which  incorporates  assessments  from 
phase  1  and  (through  game-play)  examines: 

•3  more  STA  abilities 

•5  STA  behaviors 

4.  Validate  the  system  thinking  assessment 
game  in  an  operational  setting 


STA  Game  Mockup 
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•  Identifying  talent  among  the  Army’s  new  accessions 
remains  a  priority  for  the  Army’s  cyber  occupations. 

•  The  Cyber  Test  has  been  a  helpful  solution  based  on  the 
research  conducted  to  date. 

-  There  is  the  possibility  that  if  the  Army  continues  to  use  the 
Cyber  Test,  ARI  may  eventually  invest  in  a  computerized 
adaptive  test  solution  to  enhance  test  security. 

•  Benefits  of  selection  tools  for  initial  entry  personnel 
selection: 

•  Increased  personnel  performance 

•  Increased  personnel  attitudes  (e.g.,  adjustment,  satisfaction) 

•  Increased  match  between  individual  characteristics  and  job 
requirements 

•  Reduced  attrition 
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